The invention relates generally to computer graphics, and more particularly to acquiring images of three-dimensional physical objects to generate 3D computer graphics models that can be rendered in realistic scenes using the acquired images.
Three-dimensional computer graphics models are used in many computer graphics applications. Generating 3D models manually is time consuming, and causes a bottleneck for many practical applications. Besides the difficulty of modeling complex shapes, it is often impossible to replicate the geometry and appearance of complex objects using prior art parametric reflectance models.
Not surprisingly, systems for generating 3D models automatically by scanning or imaging physical objects have greatly increased in significance. An ideal system would acquire the shape and appearance of an object automatically, and construct a detailed 3D model that can be placed in an arbitrary realistic scene with arbitrary novel illumination.
Although there has been much recent work towards this goal, no system to date fulfills all of these requirements. Many systems, including most commercial systems, focus on capturing accurate shape, but neglect to acquire an accurate appearance. Other methods capture reflectance properties of 3D objects and fit these properties to parametric bi-directional reflectance distribution functions (BRDFs). However, those methods fail for complex anisotropic BRDFs and do not model important appearance effects, such as inter-reflections, self-shadowing, translucency, sub-surface light scattering, or refraction.
There have also been a number of image-based methods for acquiring and representing complex objects. But they either lack a 3D shape model, assume accurate 3D geometry, do not allow rendering the objects under novel arbitrary illumination, or are restricted to a single viewpoint. All of these systems require substantial manual involvement.
There are many methods for acquiring high-quality 3D shape from physical objects, including contact digitizers, passive stereo depth-extraction, and active light imaging systems. Passive digitizers are inadequate where the object being digitized does not have sufficient texture. Nearly all passive methods assume that the BRDF is Lambertian, or does not vary across the surface.
Magda et al., in xe2x80x9cBeyond Lambert: Re-constructing Surfaces with Arbitrary BRDFs.xe2x80x9d Proc. of IEEE International Conference on Computer Vision ICCV, 2001, described a stereopsis method that uses the Helmholtz reciprocity to extract depth maps from objects with arbitrary BRDFs. However, their method is not robust for smooth objects. In addition, their method does not take inter-reflections and self-shadowing into account.
Active light systems, such as laser range scanners, are very popular and have been employed to acquire large models in the field, see Levoy et al. xe2x80x9cThe Digital Michelangelo Project: 3D Scanning of Large Statues,xe2x80x9d Computer Graphics, SIGGRAPH 2000 Proceedings, pp. 131-144, 2000, and Rushmeier et al. xe2x80x9cAcquiring Input for Rendering at Appropriate Levels of Detail: Digitizing a Piet""a,xe2x80x9d Proceedings of the 9th Eurographics Workshop on Rendering, pp. 81-92, 1998.
Active light systems often require a registration step to align separately acquired scanned meshes, see Curless et al., xe2x80x9cA Volumetric Method for Building Complex Models from Range Images,xe2x80x9d Computer Graphics, SIGGRAPH 96 Proceedings, pp. 303-312, 1996, and Turk et al., xe2x80x9cZippered Polygon Meshes from Range Images,xe2x80x9d Computer Graphics, SIGGRAPH 94 Proceedings, pp. 311-318, 1994. Alternatively, the scanned geometry is aligned with separately acquired texture images, see Bernardini et al., xe2x80x9cHigh-Quality Texture Reconstruction from Multiple Scans,xe2x80x9d IEEE Trans. on Vis. and Comp. Graph., 7(4):318-332, 2001.
Often, filling of gaps due to missing data is necessary as well. Systems have been constructed where multiple lasers are used to acquire a surface color estimate along lines-of-sight of the imaging system. However, those systems are not useful for capturing objects under realistic illumination. All active light systems place restrictions on the types of materials that can be scanned, as described in detail by Hawkins et al., in xe2x80x9cA Photometric Approach to Digitizing Cultural Artifacts,xe2x80x9d 2nd International Symposium on Virtual Reality, Archaeology, and Cultural Heritage, 2001.
To render objects constructed of arbitrary materials, image-based rendering can be used. Image-based representations have the advantage of capturing and representing an object regardless of the complexity of its geometry and appearance. Prior art image-based methods allowed for navigation within a scene using correspondence information, see Chen et al., xe2x80x9cView Interpolation for Image Synthesis,xe2x80x9d Computer Graphics,xe2x80x9d SIGGRAPH 93 Proceedings, pp. 279-288, 1993, and McMillan et al., xe2x80x9cPlenoptic Modeling: An Image-Based Rendering System,xe2x80x9d Computer Graphics, SIGGRAPH 95 Proceedings, pp. 39-46, 1995. Because this method does not construct a model of the 3D object, it is severely limited.
Light field methods achieve similar results without geometric information, but with an increased number of images, see Gortler et al, xe2x80x9cThe Lumigraph,xe2x80x9d Computer Graphics, SIGGRAPH 96 Proceedings, pp. 43-54, 1996, and Levoy et al., xe2x80x9cLight Field Rendering,xe2x80x9d Computer Graphics, SIGGRAPH 96 Proceedings, pp. 31-42, 1996. The best of those methods, as described by Gortler et al., include a visual hull of the object for improved ray interpolation. However, those methods use static illumination, and cannot accurately render objects into novel arbitrary realistic scenes.
An intermediate between purely model-based and purely image-based methods uses view-dependent texture mapping, see Debevec et al., xe2x80x9cModeling and Rendering Architecture from Photographs: A Hybrid Geometry-and Image-Based Approach,xe2x80x9d Computer Graphics, SIGGRAPH 96 Proceedings, pp. 11-20, 1996, Debevec et al., xe2x80x9cEfficient View-Dependent Image-Based Rendering with Projective Texture-Mapping,xe2x80x9d Proceedings of the 9th Eurographics Workshop on Rendering, pp. 105-116, 1998, and Pulli et al., xe2x80x9cView-Based Rendering: Visualizing Real Objects from Scanned Range and Color Data,xe2x80x9d Eurographics Rendering Workshop 1997, pp. 23-34, 1997. They combine simple geometry and sparse texture data to accurately interpolate between images. Those methods are effective despite their approximate 3D shapes, but they have limitations for highly specular surfaces due to the relatively small number of texture maps.
Surface light fields can be viewed as a more general and more efficient representation of view-dependent texture maps, see Nishino et al., xe2x80x9cEigen-Texture Method: Appearance Compression based on 3D Model,xe2x80x9d Proc. of Computer Vision and Pattern Recognition, pp. 618-624, 1999, Miller et al., xe2x80x9cLazy Decompression of Surface Light Fields for Precomputed Global Illumination,xe2x80x9d Proceedings of the 9th Eurographics Workshop on Rendering, pp. 281-292, 1998, Nishino et al., xe2x80x9cAppearance Compression and Synthesis based on 3D Model for Mixed Reality,xe2x80x9d Proceedings of IEEE ICCV ""99, pp. 38-45, 1999, Grzeszczuk, xe2x80x9cAcquisition and Visualization of Surface Light Fields,xe2x80x9d Course Notes, SIGGRAPH 2001, 2001, and Wood et al., xe2x80x9cSurface Light Fields for 3D Photography,xe2x80x9d Computer Graphics, SIGGRAPH 2000 Proceedings, pp. 287-296, 2000. Wood et al. store surface light field data on accurate high-density geometry, whereas Nishino et al. use a coarser triangular mesh for objects with low geometric complexity.
Surface light fields are capable of reproducing important global lighting effects, such as inter-reflections and self-shadowing. Images generated with a surface light field usually show the object under a fixed lighting condition. To overcome this limitation, inverse rendering methods estimate the surface BRDF from images and geometry of the object.
To achieve a compact BRDF representation, most methods fit a parametric reflection model to the image data, see Lensch et al., xe2x80x9cImage-Based Reconstruction of Spatially Varying Materials,xe2x80x9d Proceedings of the 12th Eurographics Workshop on Rendering, 2001, Sato et al.,xe2x80x9d Object Shape and Reflectance Modeling from Observation,xe2x80x9d Computer Graphics, SIGGRAPH 97 Proceedings, pp. 379-387, 1997, and Yu et al., xe2x80x9cInverse Global Illumination: Re-covering Reflectance Models of Real Scenes from Photographs,xe2x80x9d Computer Graphics, SIGGRAPH 99 Proceedings, pp. 215-224, 1999.
Sato et al. and Yu et al. assume that the specular part of the BRDF is constant over large regions of the object, while the diffuse component varies more rapidly. Lensch et al. fit a Lafortune BRDF model to each point on the object surface. Simple parametric BRDFs, however, are incapable of representing the wide range of reflections seen in real scenes. As described by Hawkins et al., objects featuring glass, fur, hair, cloth, leaves, or feathers are very challenging or impossible to represent this way.
An alternative method uses image-based, non-parametric representations for object reflectance, see Marschner et al., xe2x80x9cImage-based BRDF Measurement Including Human Skin,xe2x80x9d Proceedings of the 10th Eurographics Workshop on Rendering, pp. 139-152, 1999. They use a tabular BRDF representation and measure the reflectance properties of convex objects using a digital camera. Their method is restricted to objects with a uniform BRDF, and they incur problems with geometric errors introduced by 3D range scanners. Image-based relighting can also be applied to real human faces by assuming that the surface reflectance is Lambertian, see Georghiades et al., xe2x80x9cIllumination-Based Image Synthesis: Creating Novel Images of Human Faces under Differing Pose and Lighting,xe2x80x9d IEEE Workshop on Multi-View Modeling and Analysis of Visual Scenes, pp. 47-54, 1999.
More recent approaches use image databases to re-light models of objects from a fixed viewpoint without acquiring a full BRDF, see Debevec et al., xe2x80x9cAcquiring the Reflectance Field of a Human Face,xe2x80x9d Computer Graphics, SIGGRAPH 2000 Proceedings, pp. 145-156, 2000, Hawkins et al., xe2x80x9cA Photometric Approach to Digitizing Cultural Artifacts,xe2x80x9d 2nd International Symposium on Virtual Reality, Archaeology, and Cultural Heritage, 2001, Koudelka et al., xe2x80x9cImage-based Modeling and Rendering of Surfaces with Arbitrary BRDFs, xe2x80x9cProc. of Computer Vision and Pattern Recognition, 2001, and Malzbender et al., xe2x80x9cPolynomial Texture Maps, Computer Graphics, SIGGRAPH 2001 Proceedings, pp. 519-528, 2001.
They use a light stage with fixed camera positions and a rotating light to acquire the reflectance field of a human face, or of cultural artifacts. The polynomial texture map system described by Malzbender et al. uses a similar technique for objects with approximately planar geometry and diffuse reflectance properties. Koudelka et al. use essentially the same method as Debevec et al. to render objects with arbitrary BRDFs. Those reflectance field approaches are limited to renderings from a single viewpoint. In Debevec et al., a parametric model of skin reflectance is fit to the data to synthesize views from arbitrary directions. As described by Kaudelka et al., that approach does not generalize to complex materials.
In the prior art, a surface reflectance field of an object is defined as the radiant light from a surface under every possible incident field of illumination. It is important to note that, despite the word reflectance, the surface reflectance field captures all possible surface and lighting effects, including surface texture or appearance, refraction, dispersion, subsurface scattering, and non-uniform material variations.
Zongker et al. describe techniques of environment matting to capture mirror-like and transparent objects, and to correctly composite them over arbitrary backgrounds, see Zongker et al., xe2x80x9cEnvironment Matting and Compositing. In Computer Graphics, SIGGRAPH 1999 Proceedings, pp. 205-214, August 1999. Their system is able to determine the direction and spread of the reflected and refracted rays by illuminating a shiny or refractive object with a set of coded light patterns. They parameterize surface reflectance into 2D environment mattes. Extensions to environment matting include a more accurate capturing method and a simplified and less accurate procedure for real time capture of moving objects. However, their system only captures environment mattes for a fixed viewpoint, and they do not reconstruct the 3D shape of the object.
Therefore, it is desired to automatically generate computer graphics models from objects made of arbitrary materials under various lighting conditions, so that the models can be rendered in realistic scenes with arbitrary lighting.
The invention provides a system and method for acquiring and rendering high quality graphical models of physical objects, including objects constructed of highly specular, transparent, or fuzzy materials, such as fur and feathers, that are difficult to handle with traditional scanners. The system includes a turntable, an array of digital cameras, a rotating array of directional lights, and multi-color backlights. The system uses the same set of acquired images to both construct a model of the object, and to render the object for arbitrary lighting and points of view.
The system uses multi-background matting techniques to acquire alpha mattes of 3D objects from multiple viewpoints by placing the objects on the turntable. The alpha mattes are used to construct an opacity hull. An opacity hull is a novel shape model with view-dependent opacity parameterization of the surface hull of the object. The opacity hull enables rendering of complex object silhouettes and seamless blending of objects into arbitrary realistic scenes.
Computer graphics models, according to the invention, are constructed by acquiring radiance, reflection, and refraction images. During rendering, the models can also be lighted from arbitrary directions using a surface reflectance field, to produce a purely image-based appearance representation. The system is unique in that it acquires and renders surface appearance under varying illumination from arbitrary viewpoints. The system is fully automatic, easy to use, has a simple set-up and calibration. The models acquired from objects can be accurately composited into synthetic scenes.